Skip to content

feat(gateway): keep a plugin entity's freeze frame across a gateway restart - #680

Draft
bburda wants to merge 17 commits into
fix/opcua-demo-runs-a-fault-managerfrom
fix/persist-entity-freeze-frames
Draft

bburda wants to merge 17 commits into
fix/opcua-demo-runs-a-fault-managerfrom
fix/persist-entity-freeze-frames

Conversation

@bburda

@bburda bburda commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

A plugin entity's freeze frame lived only in gateway memory. After a restart the gateway read the plant again and served those values under a new captured_at, as if they were the values at confirmation.

The core freeze-frame capture now writes its frames to a SQLite store when entity_freeze_frame.storage.path is set. When the path is empty and triggers.storage.path is set, the frames go to entity_freeze_frames.db in the same directory. Each row records the fault manager's occurrence of the fault (occurrence_count and first_occurred).

At start the gateway asks the fault manager once for every fault in every status, muted faults included, and judges each reloaded frame:

  • same occurrence: the frame is served as it was captured, with its original captured_at;
  • another occurrence: the frame is dropped with a warning; a fault that is still confirmed is then read by the startup catch-up and marked capture_origin: startup;
  • fault not held by the fault manager: the frame is dropped, but only when the fault list is not empty, because an empty list can also mean the fault storage is unreachable.

A reloaded frame is served only after that judgement. A live confirmation of a new occurrence whose read gets no values drops the previous occurrence's frame, so the fault has no frame. A recapture counts as the newest frame for the 256-fault bound.

In subscription mode the OPC UA plugin now reads every subscribed point once at start. Before, GET /data and the startup capture saw null values until the first publish, which can be a whole publishing interval later. After a reconnect the plugin keeps its last known values until the first publish.

The OpenPLC tank demo keeps the gateway state on the named volume ros2-medkit-opcua-state across stop.sh and start.sh: the entity frames and the fault manager's faults.db.

Stacked on #790.


Issue


Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

Testing

The new freeze-frame-restart job runs run_freeze_frame_restart_tests.sh. It restarts a real gateway and fault manager against the OPC UA test server and checks:

  • a frame of the same occurrence keeps its values and captured_at;
  • a fault that is cleared and reported again while the gateway is down gets a startup frame;
  • a fault that is cleared, confirmed again and cleared again while the gateway is down serves no frame of the first occurrence;
  • an unreachable OPC UA server at start leaves no stale frame;
  • an empty fault manager drops nothing, and a fault manager that no longer holds a fault drops its frame;
  • subscription mode with a 60 s publishing interval serves values right after start;
  • each entity_freeze_frame.storage.path case: a path, empty with triggers.storage.path, both empty, and a path that cannot be opened.

The demo-faults job also checks that the demo's frame keeps its captured_at across stop.sh and start.sh. The workflow now also runs when the core capture, its store or their wiring in gateway_node change.


Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

A plugin entity's freeze-frame lived in gateway memory only, so a restart
replaced it with a read of the plant as it was at the restart, under a new
captured_at. The values frozen when the fault confirmed were gone.

The core capture now takes an optional EntityFreezeFrameStore through its
Hooks, next to it in the core and free of ROS. Every capture writes the
fault's frames through to the store, replacing the code's rows, and an
evicted fault loses its rows too. The stored frames are loaded on
construction, before any capture: past the retained-frame bound the newest
codes are kept and the older rows deleted, unreadable rows are skipped and
counted. A reloaded frame is served as it was captured, captured_at and
capture_origin included.

The startup catch-up first drops reloaded frames whose code the fault
manager no longer holds in any status, asked through a new
KnownFaultCodeLister (nothing is dropped when it cannot answer). A reloaded
frame whose fault has a newer first_occurred than the frame belongs to an
earlier occurrence: it is read again before the bound counts anything,
replaced and marked startup when the read answers, and dropped with a
warning naming the fault and the entities when it does not. Each such code
is asked once per catch-up, and the truncation warning names up to ten
faults left without a frame.

SqliteEntityFreezeFrameStore keeps the frames in a file. The node opens it
from the new entity_freeze_frame.storage.path, else entity_freeze_frames.db
next to triggers.storage.path; with neither, or a file it cannot open, the
frames stay in memory and the log says so. StandingFault carries
first_occurred_ns, parsed from the ListFaults reply with a range check.
stop.sh removes the gateway container, so faults.db, the rosbags and the
entity freeze-frames went with its writable layer. start.sh now mounts the
named volume ros2-medkit-opcua-state (OPCUA_DEMO_STATE_VOLUME overrides it)
at /var/lib/ros2_medkit, stop.sh keeps it and says how to purge it, and the
demo parameters put the freeze-frame store on it. A fault raised before
stop.sh then still serves the values frozen when it confirmed after the
next start.sh.

The demo-faults job checks exactly that: it confirms PLC_HIGH_TEMP, runs
stop.sh and start.sh, and requires the same captured_at and no
capture_origin once a second alarm shows the startup catch-up is done.
run_freeze_frame_restart_tests.sh boots test_alarm_server and one gateway
container that runs fault_manager_node and gateway_node with the state on a
named volume, and drives the server's values through its stdin CLI. It
checks that a replaced gateway container serves the frame captured at
confirmation; that a fault cleared and reported again while gateway_node was
down is read again on start and marked capture_origin: startup; that such a
frame is dropped with a warning when the server is unreachable, and when the
fault manager restarts with empty storage; and each
entity_freeze_frame.storage.path case. A check that something did not
happen waits for a sentinel fault the drain loop frames after the startup
catch-up.

The workflow runs it as its own job, and a change to the core freeze-frame
capture, its store or the ROS adapter now triggers the workflow.
…ction

At the retained-frame bound, a fault confirmed again kept its old place in
the eviction order, so the next new fault evicted the frame just captured.
After a restart the frames are ordered by captured_at, so a different fault
went instead. A recapture now moves its fault to the newest end, which is
the order a reload rebuilds.
The SQLite store skipped a row whose frame column is not JSON before the
capture could see it, so the "unreadable row(s) skipped" warning counted
only rows that were JSON but not shaped like a frame. load_all now returns
such a row with a null frame, and the capture skips and counts both kinds.
A fault manager whose storage connection is lost answers ListFaults with an
empty list as a success. The startup prune read that as "every fault is
gone" and deleted every reloaded frame and its rows for good. The prune now
runs only when the fault manager lists at least one fault. An empty list
drops nothing and logs one info line. A frame whose fault the fault manager
no longer holds then leaves by eviction or by the next capture of its code.
At startup a reloaded frame counted as stale when the fault manager's
first_occurred was later than the frame's captured_at. That compared the
clocks of two processes, so a clock skew or step either re-read a frame of
the same occurrence or kept the frame of an earlier one. Only confirmed
faults were judged, so a fault that cleared, confirmed again and cleared
again while the gateway was down served the first occurrence's frame.

Each stored row now carries the fault manager's occurrence of the fault at
capture: occurrence_count and first_occurred, which the fault manager
changes only when a cleared fault fails again. A confirm-edge capture takes
them from the event, a catch-up capture from the ListFaults reply, both in
the reply's encoding so one occurrence gives equal values. At startup every
reloaded code is looked up in the all-status fault list:

- same occurrence: kept, no read;
- another occurrence, confirmed: read again and marked startup;
- another occurrence, not confirmed: discarded with a warning that names
  the fault, the entities and the reason;
- a row or a listed fault without an occurrence: kept.

The known-fault lister now returns the listed faults with their status and
occurrence, parsed by standing_faults_from_list_reply.
…eviction age

The restart section of the snapshots tutorial, the
entity_freeze_frame.storage.path row and the gateway README and design
notes now say that a reloaded frame is judged by the occurrence stored with
it, that an empty fault list drops nothing, and that the bound evicts the
frame captured longest ago, counting a fault confirmed again from its new
capture.
…empty fault list

The freeze-frame restart script now also shows, against the real gateway
and fault manager:

- a fault cleared while the gateway runs keeps its frame and captured_at
  across a restart;
- a fault cleared, confirmed again and cleared again while gateway_node is
  down does not serve the first occurrence's frame, and the gateway logs a
  warning that names the fault;
- a fault manager with an empty fault list drops no stored frame, and one
  that holds an unrelated fault drops the frame of a fault it does not hold.

The fault list helper asks for every status, so a cleared fault is seen.
The start used two ListFaults calls, a confirmed-only list for the catch-up
and an all-status list for the reloaded frames, plus a pass that read stale
frames again. Taken at different moments and with different filters they
could disagree: a muted confirmed fault kept its stale frame for a re-read
that never came, a fault that cleared between the calls was discarded and
then read again from the older list, and a failed first call with a working
second one kept a stale confirmed frame. Reloaded frames were also served
before any of this had run.

The capture now makes one call over every status, muted faults included.
That answer judges each reloaded frame: the same occurrence is kept with no
read, another occurrence is discarded with a warning that names the fault,
the entities, both occurrence counts and the status, and a fault the list
does not hold is dropped unless the list is empty. The catch-up then reads
every confirmed fault that is not muted and has no frame, so a stale
confirmed fault gets its startup frame through the normal path. A muted
fault is not framed at start, as the fault manager publishes no confirm
event for it. frames_for serves no reloaded frame until the judgement has
run or the lister has given up.

The frames dropped for faults the fault manager no longer holds are logged
as one warning that names up to 10 codes.

The known-fault lister hook is gone; the standing-fault lister returns every
fault with its status, occurrence and muted flag, and a fault without a
status counts as confirmed.
REQ_INTEROP_088, the entity_freeze_frame rows in the server configuration,
the snapshots tutorial and the gateway README and design notes now say that
a fault already confirmed at start is read and marked startup only when no
frame of the same occurrence was reloaded, that one fault list of every
status judges the reloaded frames before any is served, that a frame of
another occurrence is discarded and a confirmed fault then framed by the
catch-up, that a muted fault is not framed at start, and that frames of
faults the fault manager no longer holds are dropped with a warning that
names them.
The freeze-frame restart script follows the start that one fault list
decides: a frame from an earlier occurrence is discarded with a warning that
names the fault, the entity, both occurrence counts and the status, a
confirmed fault then gets its startup frame from the catch-up, and the frames
of faults the fault manager no longer holds go with one warning that names
them.

A new scenario runs the plugin in subscription mode with a 60 s publishing
interval: right after the gateway is up, GET /data must serve the current
value within 15 s, and a stale frame of a confirmed fault must be replaced at
start by a startup frame that holds the current value.

The plugin README describes the scenarios and quotes the warning.
In subscription mode the plugin's cache stayed empty until the
subscription's first publish, up to one subscription_interval_ms after the
subscriptions were created. Until then GET /data served nulls and a startup
freeze-frame read found no values. With a 60 s interval the first values
came 62 s after the poller started.

After the plugin creates its subscriptions, at start and again after each
reconnect, it now reads every subscribed point once, so the cache holds
current values at once. The README says so for subscription_interval_ms.
A frame held for a fault, reloaded and judged valid or captured in this
life, stayed when the fault confirmed as its next occurrence and the read
of its entity gave nothing (PLC down, cold cache). The detail of the new
occurrence then served the previous occurrence's values and captured_at,
and the stored rows still named the previous occurrence.

The capture now keeps the occurrence of every held frame. A confirm whose
occurrence differs from it and whose read gives no frame drops the held
frame and its rows and logs a warning that names the fault and says this
occurrence has no freeze-frame. A confirm of the same occurrence, or one
that carries no occurrence, keeps the frame as before.
gateway_node and the fault scope code wire the entity freeze-frame capture
into the gateway, yet a change to them did not trigger the OPC UA workflow
that runs the freeze-frame restart test. Both paths lists now name them,
and the header comment says so.
- The REST snapshot types: a fault confirmed before the gateway started is
  read at start and marked startup only when no frame of the same
  occurrence was reloaded.
- connected: false appears only when the plugin's payload carries a link
  flag; the OPC UA plugin's DataProvider payload carries none.
- A reloaded frame is served only after the start has judged it, except
  when the host gives no lister or the lister throws or answers with an
  empty list: then the frames are kept and served unjudged.
- The store file belongs to one gateway. Two gateways must not share it,
  since one gateway's start drops the other's frames.
The read after creating the subscriptions also ran when they were created
again after a reconnect. There the alarm event pump keeps iterating the
client while the poll thread reads, so a publish handled inside the read
could deliver a newer value that the older read result then overwrote,
with nothing to correct it until the point changed.

The read now runs once at start, before the poll thread and the event pump
exist. After a reconnect the cache keeps the last known values until the
first publish. The README row for subscription_interval_ms says so.
@bburda
bburda force-pushed the fix/persist-entity-freeze-frames branch from b4bf815 to 016a68e Compare November 17, 2026 08:49
@bburda
bburda changed the base branch from main to fix/opcua-demo-runs-a-fault-manager November 17, 2026 08:51
@bburda
bburda added this pull request to stack #791 November 17, 2026 08:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant